Image-processing method

ABSTRACT

An image-processing method is provided with the first step of dividing an input image having n(n&gt;1) gray scales into a plurality of matrixes, the second step of carrying out at least either a resolution-converting process or a variable magnification process for each of the divided matrixes, by using a hierarchical neural network that can execute a learning process for each input image, and the third step of outputting the image processed in the second step as an output image having n gray scales. Thus, the weights adjustment of the network can be carried out on each input image whatever image is inputted thereto; therefore, it is possible to always provide an optimal converting process.

FIELD OF THE INVENTION

The present invention relates to an image-processing method in whichwith respect to an input image with n(n>1) gray scales, a conversionprocess is applied so as to convert the resolution and magnification ofthe image and to obtain an output image with n gray scales.

BACKGROUND OF THE INVENTION

Conventionally, various methods have been proposed as image-processingmethods which apply conversion processes to an input image with n(n−1)gray scales so as to convert the resolution and magnification of theimage and obtain an output image with n gray scales.

With respect to methods for converting the resolution of an image, asimple interpolation method, which uses pixel values prior to aninterpolation as peripheral interpolated pixel values after theinterpolation has been carried out as shown in FIG. 7(a), and an averagemethod, which uses the average value of pixel values prior to aninterpolation as pixel values after the interpolation has been carriedas shown in FIG. 7(b), have been known.

Moreover, a linear interpolation method, which connects respective pixelvalues with straight lines prior to an interpolation and which usesvalues on the straight lines as interpolation values, for example, asshown in FIG. 7(c), is known as another method for converting theresolution of an image. At present, this linear interpolation method ismost widely adopted.

With respect to methods for converting the magnification of an image, asimple interpolation method, which uses pixel values prior to aninterpolation as peripheral varied magnification pixel values after theinterpolation has been carried out as shown in FIG. 8(a), and an averagemethod, which uses the average value of pixel values prior to aninterpolation as varied magnification pixel values after theinterpolation has been carried out as shown in FIG. 8(b), have beenknown.

Moreover, a linear interpolation method, which connects respective pixelvalues with straight lines prior to an interpolation and which usesvalues on the straight lines as interpolation values, for example, asshown in FIG. 8(c), is known as another method for converting theresolution of an image. At present, this linear interpolation method ismost widely adopted.

The above-mentioned methods, however, cause a problem in which an image,which has been subjected to the resolution-converting process or thevariable-magnification process, has blurred edge portions or burredslanting lines, resulting in a lack of smoothness.

In order to solve this problem, for example, “Image-InterpolatingApparatus”, disclosed in Japanese Laid-Open Patent Publication12486/1994 (Tokukaihei 6-12486), carries out a non-linear enlargingprocess on an input image by using a neural network so that the imagethat appears after the process has sharpness in its edge portions andhas its burred slanting lines masked, resulting in a smooth image.

The enlarging process in the image-interpolating apparatus in theabove-mentioned laid-open patent publication is explained as follows:First, four pixel regions including a focused pixel, its right-handadjacent pixel, its lower adjacent pixel and its diagonal lower adjacentpixel are extracted from a binary image. Next, provided that the valuesof the pixels in the four pixel regions are altered, that is, providedthat the focused pixel is enlarged K times in the main scanningdirection and L times in the sub-scanning direction, pixels in theenlarged region are statistically analyzed to find such pixels in theenlarged region as to be smoothly connected to the peripheral pixels,and by using the results as teaching signals and the values of therespective pixels in the four pixel regions as input signals, a learningoperation for the neural network is preliminarily carried out. Then, theenlarging process of the inputted binary image is carried out by usingthe neural network after having been subjected to the learningoperation.

However, in the enlarging process by the image-interpolating apparatusas disclosed in the above-mentioned laid-out patent publication, sincethe neural network, which has been subjected to a learning operationusing statistical data that has been preliminarily provided, is adopted,weights at the connecting sections of the neural network after thelearning operation has been carried out are fixed. For this reason, fineadjustments of the weights cannot be made by applying re-learningoperations; thus, the resulting problem is that in the case ofinsufficient statistical data, it is not possible to obtain smoothlyenlarged images depending on various input images.

SUMMARY OF THE INVENTION

The objective of the present invention is to provide an image-processingmethod by which, irrespective of an image to be inputted, it becomespossible to obtain an output image having smooth edge portions andslanting lines even when the resolution and magnification of the imageis converted.

In order to achieve the above-mentioned objective, the image-processingmethod of the present invention divides an input image having n(n>1)gray scales into a plurality of matrixes, carries out at least either aresolution-converting process or a variable magnification process foreach of the divided matrixes, by using a hierarchical neural networkthat can execute a learning process for each input image, and outputs aprocessed image having n gray scales as an output image.

With the above-mentioned arrangement, since the resolution-convertingprocess and variable magnification process are carried out on each ofthe divided matrixes of the input image by using the hierarchical neuralnetwork that can execute a learning process for each input image, itbecomes possible to sharpen edge portions of the output image and alsoto make slanting lines thereof smooth without burs. Moreover, since thehierarchical neural network can carry out a learning process on eachinput image, the learning process is applicable at any desired time.Thus, the weights adjustment of the connecting sections of the neuralnetwork can be carried out on each input image on a real-time basis;therefore, irrespective of images to be inputted, the relationshipbetween the input image and output image can be optimized.

As described above, since at least either the resolution-convertingprocess or the variable magnification process is carried out on each ofthe divided matrixes of the input image by using the hierarchical neuralnetwork which can execute a learning process on a real-time basis, theweights of the network is adjusted for each matrix; thus, it becomespossible to always carry out an optimal converting process.

Therefore, irrespective of images to be inputted, it is possible toeliminate the blurred state of the edge portions and burs in theslanting lines which are problems caused by the converting processes,and consequently to make the image after the conversion smooth, therebyimproving the quality of the converted image.

For a fuller understanding of the nature and advantages of theinvention, reference should be made to the ensuing detailed descriptiontaken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram showing an image-processingapparatus to which an image-processing method of the present inventionis applied.

FIG. 2 is an explanatory drawing that shows a gray-scale curved surface.

FIG. 3 is a schematic drawing that shows a neural network of theback-propagation type that is a hierarchical neural network applied tothe image-processing apparatus of FIG. 1.

FIG. 4 is a graph shows a sigmoid function.

FIG. 5 is a schematic drawing that shows a fuzzy neural network that isa hierarchical neural network applied to the image-processing apparatusof FIG. 1.

FIG. 6 is a graph that shows a membership function.

FIG. 7(a) is an explanatory drawing that indicates aresolution-converting method using the conventional simple interpolationmethod.

FIG. 7(b) is an explanatory drawing that indicates aresolution-converting method using the conventional average method.

FIG. 7(c) is an explanatory drawing that indicates aresolution-converting method using the conventional linear interpolationmethod.

FIG. 8(a) is an explanatory drawing that indicates avariable-magnification processing method using the conventional simpleinterpolation method.

FIG. 8(b) is an explanatory drawing that indicates avariable-magnification processing method using the conventional averagemethod.

FIG. 8(c) is an explanatory drawing that indicates avariable-magnification processing method using the conventional averagemethod.

DESCRIPTION OF THE EMBODIMENT

The following description will discuss one embodiment of the presentinvention.

As illustrated in FIG. 1 an image-processing apparatus, which uses animage-processing method of the present embodiment, is provided with animage-data input device (image-data input means) 1, an image-dataconversion device 2, and an image-data output device (image-data outputmeans) 3. When an original image having n(n>1) gray scales is inputtedthereto, the image-processing apparatus carries out converting processesof resolution and magnification on the original image so that aconverted image having n gray scales is outputted.

In other words, the image-processing apparatus outputs the image that isobtained by applying converting processes to the input image inaccordance with the following steps:

First, the image-data input device 1 reads an original image having ngray scales as image data by using an image-reading means such as ascanner, not shown.

Next, the image-data conversion device 2 carries out convertingprocesses of resolution and magnification on the image data that hasbeen read by the image-data input device 1. The converting processes inthe image-data conversion device 2 will be described later in detail.

Finally, the image-data output device 3 stores the image data whoseresolution and magnification have been converted by the image-dataconversion device 2 in a storage means such as a memory. Thereafter, theimage-data output device 3 carries out an output process on the imagedata that has been stored in the storage means, and the processed imageis outputted as a converted image having n gray scales.

The following description will discuss the image-data conversion device2 in more detail.

In order to carry out the converting processes of resolution andmagnification on the inputted original image, the image-data conversiondevice 2 is provided with an image-data division circuit 4 which servesas an image-data division means, and an existingpixel-position-information/pixel-value extracting circuit (existingpixel-position information/pixel-value extracting means) 5, anexisting-pixel-value learning circuit (learning means) 6, aconverted-pixel-position input circuit (converted-pixel position-inputmeans) 7, a converted-pixel-value arithmetic circuit(converted-pixel-value arithmetic means) 8 and a converted-pixel-valueoutput circuit (converted-pixel-value output means) 9, which serve as aconversion-processing means.

The image-data division circuit 4 is designed to divide the image data(hereinafter, referred to as input image data) that has been read by theimage-data input device 1 into matrixes in accordance with thepositional information of respective pixels.

In this case, the size of each of the divided matrixes is basically setsmaller than the input image data; and when simplicity and high speedsof arithmetic operations are taken into consideration, the longitudinaland lateral sizes of each matrix are preferably set to divisors of thelongitudinal and lateral pixel numbers of the input image data. Here, inthe present embodiment, the size of input data is set to 255×255 pixels,and the size of each matrix is set to 3×3 pixels. Further, theresolution of the input image data is set to 300 dpi×300 dpi(longitudinal-lateral) with gray scales of n=256.

The existing pixel-position-information/pixel-value extracting circuit 5is designed to extract positional information of each of the existingpixels contained in the input image data (hereinafter, referred to aspartial image data) that has been divided by the image-data divisioncircuit 4 into the matrix with the 3×3 pixel size and a pixel value ateach position, as input data and teaching data for a hierarchical neuralnetwork, and stores the resulting data temporarily in a memory, notshown. Additionally, in the case of a variable magnification process,the positional information that exists after the variable magnificationprocess is used as positional information of respective pixels to beextracted by the existing pixel-position-information/pixel-valueextracting circuit 5.

The above-mentioned pixel value is a value indicating the gray scale,such as a concentration value and luminance value in each pixel of theimage data. Further, the teaching data, obtained from the pixel value,is data that is supplied as a target value so that the hierarchicalneural network can output a correct output value in response to eachinput value. In other words, the hierarchical neural network variesweights at the connecting sections of the network so that the outputvalue becomes closer to the teaching data.

The existing-pixel-value learning circuit 6 gives the positionalinformation of each pixel and the pixel value at each position that havebeen stored in the memory by the existingpixel-position-information/pixel-value extracting circuit 5 to thehierarchical neural network as input data and teaching data, therebyallowing the neural network to learn. The number of the learningprocesses carried out by the existing-pixel-value learning circuit 6 isset up to 1000 at maximum. However, in the case of a learning error of5%, the learning process is completed even if the number of the learningprocesses has not reached 1000 times. The learning method will bedescribed in detail later.

In the present embodiment, an explanation will be given of cases whereina neural network of the back-propagation type and a fuzzy neural networkare used as the above-mentioned hierarchical neural network.Additionally, the detailed explanations of the respective hierarchicalneural networks will be given later together with the above-mentionedlearning method.

When the hierarchical neural network is subjected to a learning processin the existing-pixel-value learning circuit 6, a gradation curvedsurface, which is formed by connecting existing pixel values withnon-linear curves, is obtained as shown in FIG. 2. In FIG. 2, blackcircles  represent existing pixels, white circles ◯ representinterpolated pixels after conversion, x and y represent the x-axis andthe y-axis of the respective input image data, and the z-axis representsthe pixel value. Thus, FIG. 2 shows the results of aresolution-converting process that was carried out so as to double theresolution of the input image data with respect to the x-axis and they-axis.

The converted-pixel position input circuit 7 inputs positionalinformation of the respective pixels after the converting process to thehierarchical neural network that has been subjected to the learningprocess in the existing-pixel-value learning circuit 6. The coordinatesof each white circle ◯ in FIG. 2, that is, the values of x and y,correspond to positional information of each pixel after the convertingprocess.

The converted-pixel-value arithmetic circuit 8 uses the hierarchicalneural network to which the positional information has been inputted bythe converted-pixel position input circuit 7, and calculates and obtainspixel values at the respective positions. The results, found by theconverted-pixel-value arithmetic circuit 8, represent the portionsindicated by the white circles ◯, and the value z of each white circle ◯gives an interpolated pixel value.

The converted-pixel-value output circuit 9 allows the memory to storethe converted pixel values that have been found by the arithmeticoperations in the converted-pixel-value arithmetic circuit 8, and thenoutputs the values to the image-data output device 3 as image data afterthe converting process.

Here, referring to FIGS. 3 through 6, an explanation will be given ofthe two types of hierarchical neural network that are used in theexisting-pixel-value learning circuit 6.

First, the following description will discuss a neural network of theback-propagation type. As illustrated in FIG. 3, this neural network hastwo inputs and one output, and is constituted by three layers includingan input layer 11, an intermediate layer 12 and an output layer 13. Inthe above-mentioned neural network, the two input items are pieces ofpositional information of each pixel, and one output item is a pixelvalue at the inputted pixel position.

The input layer 11 is provided with two nodes, that is, a node A1 towhich the input value X1 is inputted and a node A2 to which the inputvalue X2 is inputted. The reference numerals X1 and X2 represent thepositional information of each pixel; that is, X1 represents thepositional information on the x-axis of each pixel, and X2 representsthe positional information on the y-axis of each pixel.

The intermediate layer 12 is provided with nine nodes D1 through D9, andthese nodes D1 through D9 are connected to the node A1 of the inputlayer 11 with weights W11 through W19, and also connected to the node A2of the input layer 11 with weights W21 through W29. These weights W11through W19 and W21 through W29 are adjusted through learning processeswhich will be described later.

Additionally, the number of nodes of the intermediate layer 12 isdefined as the particular number of nodes in which, when learningprocesses were carried out by increasing the number of nodes in theintermediate layer 12 successively one by one from 1 by the use ofsample image data, the learning process was carried out most precisely.Therefore, the number of nodes in the intermediate layer 12 is notnecessarily limited to nine as is adopted in the present embodiment, andmay be set at a desired number of nodes in light of the learningprecision.

The output layer 13 is provided with one node E1, and the node E1 isconnected to the nodes D1 through D9 of the intermediate layer 12 withweights V11 through V91. The weights V11 through V91 are determinedthrough learning processes which will be described later, and its outputis released as an output value Y1.

The following description will discuss a learning method of the neuralnetwork of the back-propagation type that is constructed as describedabove.

First, an explanation will be given of arithmetic operations in theforward direction from the input layer 11 to the output layer 13.Positional information of partial image data is first inputted to theinput layer 11 as input data (X1·X2), and is outputted to theintermediate layer 12, as it is. In other words, this process is shownas follows:

Ii=Xi [i=1,2]  (1)

Here, Ii represents an output value from each node of the input layer11; and, for example, the output value of the node A1 is I1, and theoutput value of the node A2 is I2. Further, Xi represents input dataindicating positional information of each pixel; and, for example, X1represents input data indicating the positional information on thex-axis of each pixel, and X2 represents input data indicating thepositional information on the y-axis of the pixel.

Next, in the intermediate layer 12, arithmetic operations are carriedout based on the output values from the input layer 11, and the outputvalues of the nodes D1 through D9 are determined. More specifically, thearithmetic operations are carried out in accordance with the followingequation (2) between the input layer 11 and the intermediate layer 12.

Hj=f(Ii·Wij+θ1j)  (2)

[j=1,2, . . . , 9]

Here, Hj is an output value from each node of the intermediate layer 12;and, for example, the output value of the node D1 is H1, and the outputvalue of the node D2 is H2. Moreover, Wij is a weight that indicates thedegree of connection between each node of the input layer 11 and eachnode of the intermediate layer 12; and, for example, the degree ofconnection between the node A1 of the input layer 11 and the node D1 ofthe intermediate layer 12 is W11. Furthermore, θ1j is an offset value ineach node of the intermediate layer 12.

Supposing that the input value X is represented by (Ii·Wij+θ1j) in theabove-mentioned equation (2), f(X) becomes a non-linear monotonicallyincreasing function that monotonically increases with respect to theinput value X, and a sigmoid function, for example, as shown in FIG. 4,is applied. The sigmoid function is represented by the followingequation (3): $\begin{matrix}{{f(x)} = \frac{1}{1 + ^{- x}}} & (3)\end{matrix}$

Finally, in the output layer 13, arithmetic operations are carried outbased on the output values from the intermediate layer 12, therebydetermining the output value of the node E1. More specifically, thearithmetic operations is carried out in accordance with the followingequation (4) between the intermediate layer 12 and the output layer 13:

Ok=f(Hj·Vjk+θ2k)  (4)

[k=1]

Here, Ok is the output value Y1 from the node E1 of the output layer 13.Further, Vjk is a weight that indicates the degree of connection betweeneach of the nodes D1 through D9 of the intermediate layer 12 and thenode E1 of the output layer 13; and, for example, the degree ofconnection between the node D1 of the intermediate layer 12 and the nodeE1 is V11. Furthermore, θ2k is an offset value in the node E1 of theoutput layer 13.

Supposing that the input value X is represented by (Hj·Vjk+θ2k) in theabove-mentioned equation (4), f(X) becomes a non-linear monotonicallyincreasing function that monotonically increases with respect to theinput value X, and a sigmoid function, for example, as shown in FIG. 4,is applied in the same manner as the above-mentioned equation (2).

Next, the following description will discuss arithmetic operations inthe reverse direction for learning processes.

The objective of the learning processes is to obtain an optimalinput/output relationship in the neural network, and for this reason,the weights of the connecting points in the network are finely adjustedtowards the teaching data as targets. A method for providing fineadjustments on the weights of the connecting points in the network willbe explained below.

First, the squared error between the output value of the node E1 of theoutput layer 13 and the teaching data is calculated by using thefollowing equation (5): $\begin{matrix}{{Ek} = \frac{\left( {{Tk} - {Ok}} \right)^{2}}{2}} & (5)\end{matrix}$

Here, Ek represents the squared error between the teaching data and theoutput value, and Tk represents the teaching data. In other words, theobjective of the learning processes is to make Ek smaller. Therefore, byusing the following equation (6), the effect of Ok in Ek is found bysubjecting Ek to a partial differential with Ok. $\begin{matrix}{\frac{\partial{Ek}}{\partial{Ok}} = {- \left( {{Tk} - {Ok}} \right)}} & (6)\end{matrix}$

Further, the effect of the weight of the connection Vjk between theintermediate layer 12 and the output layer 13 in Ek and the effect ofthe weight of the connection Wij between the input layer 11 and theintermediate layer 12 in Ek are found by using the following equations(7) and (8): $\begin{matrix}{\frac{\partial{Ek}}{\partial{Vjk}} = {{- \left( {{Tk} - {Ok}} \right)} \cdot {Ok} \cdot \left( {1 - {Ok}} \right)}} & (7) \\{\frac{\partial{Ek}}{\partial{ij}} = {{{Hj} \cdot \left( {1 - {Hj}} \right) \cdot \left( {- 1} \right)}{\sum\limits_{k}\left( {\frac{\partial{Ek}}{\partial{Vjk}} \cdot {Vjk}} \right)}}} & (8)\end{matrix}$

Based upon the effect in Ek found by the above-mentioned equations (7)and (8), the weights at the connections are finely adjusted by using thefollowing equations (9) and (10): $\begin{matrix}{{{Vjk}\left( {t + 1} \right)} = {{{Vjk}(t)} + {\alpha \cdot \frac{\partial{Ek}}{\partial{Vjk}} \cdot {Hj}}}} & (9) \\{{{Wij}\left( {t + 1} \right)} = {{{Wij}(t)} + {\alpha \cdot \frac{\partial{Ek}}{\partial{Wik}} \cdot {Ii}}}} & (10)\end{matrix}$

Here, α is a value indicating the degree at which the fine adjustment iscarried out, and is normally set in the range of 0.05 to 0.25. Further,t represents the number of learning processes.

Therefore, in the above-mentioned equations (9) and (10), the weights atconnections, which are to be used in the next arithmeticoperation/learning process, are obtained by adding finely adjustingvalues to the present weights (Vjk, Wij) at connections.

As described above, by revising the weights at connections through therepeated learning processes based on the above-mentioned algorithms, thesquared error between the teaching data and the output value can be madesmaller to a certain extent. Then, the learning processes are completedat the time when a predetermined number of learning processes has beenreached or when the value of the error has become not more than anacceptable value of the error. In the present embodiment, thepredetermined number of learning processes is set to 1000 times, theacceptable value of the error is set to 5%, and the learning processesare completed at the time when the number of learning processes hasreached 1000 times or when the error has become not more than 0.5%.

By the use of a neural network of the back-propagation type that hasbeen subjected to the learning processes as described above, theresolution-converting process and variable magnification process arecarried out on all the partial images of an inputted image; thus, theinputted image is subjected to the converting process, and an imagehaving a converted resolution or an image having a varied magnificationis obtained.

Next, the following description will discuss a fuzzy neural network. Asillustrated in FIG. 5, the present neural network has two inputs and oneoutput, and is constituted by five layers including an input layer 21,the former-half of a membership layer 22, the latter-half of amembership layer 23, a rule layer 24 (these three layers serving asintermediate layers), and an output layer 25. Here, the second and thirdlayers are combined into the membership layer. In the above-mentionedfuzzy neural network, two input items are pieces of positionalinformation regarding each pixel, and one output item is a pixel valueat the inputted pixel position.

In the above-mentioned fuzzy neural network, connections between nodesin the respective layers are constructed as follows:

First, the input layer 21 is constituted by two pairs of nodes A1, A2and A3, A4, each pair corresponding to each input item. Here, a constant1 is inputted to the nodes A2 and A4, and an input value X1 (positionalinformation on the x-axis) is inputted to the node A1, and an inputvalue X2 (positional information on the y-axis) is inputted to the nodeA3.

Next, in the membership layer, big, middle and small membershipfunctions as shown in FIG. 6, are provided for each of the input items.

Therefore, in the former-half of the membership layer 22, two groups offour nodes B1 through B4 and B5 through B8 are provided, and in thenodes B1 through B4, the constant 1 and the input value X1 areconnected, while in the nodes B5 through B8, the constant 1 and theinput value X2 are connected. In the latter-half of the membership layer23, two groups of three nodes C1 through C3 and C4 through C6 areprovided, and one or two of the nodes of the former-half of themembership layer 22 are connected thereto.

More specifically, the nodes C1 and C4, which serve as portions forconnecting one of the nodes of the former-half of the membership layer22, form portions constituting the big membership function, and thenodes C3 and C6 form portions constituting the small membershipfunction. Further, the nodes C2 and C5, which serve portions forconnecting two of the nodes of the former-half of the membership layer22, form portions constituting the middle membership function.

The nodes of the membership layer with the above-mentioned arrangementare inevitably provided for each input item; thus, the number of nodesfor each item is fixed. In the present embodiment, the number of nodesof the former-half of the membership layer 22 for each input item is setat four, and the number of nodes of the latter-half of the membershiplayer 23 is set at three.

Next, in the rule layer 24, nodes D1 through D3 are constructed so thatthey are subjected to logical ANDs with the respective nodes C4 throughC6 that are related to the input value X2 with respect to the node C1that is related to the input value X1 of the latter-half of themembership layer 23. In the same manner, nodes D4 through D6 areconstructed so that they are subjected to logical ANDs with therespective nodes C4 through C6 that are related to the input value X2with respect to the node C2 that is related to the input value X1, andnodes D7 through D9 are constructed so that they are subjected tological ANDs with the respective nodes C4 through C6 that are related tothe input value X2 with respect to the node C3 that is related to theinput value X1. In other words, in the rule layer 24, all thecombinations of the membership values are given with respect to the twoinput values X1 and X2; thus, fuzzy logical ANDs are obtained.

Finally, the output layer 25 is provided with one node E1 so that allthe outputs from the rule layer 24 are connected and an output value Y1,which is a pixel value at each pixel position of the input image, isreleased therefrom.

At the connecting sections between the nodes in the fuzzy neural networkhaving the above-mentioned construction, each connection has a specificweight.

Accordingly, at the connecting sections between the input layer 21 andthe former-half of the membership layer 22, center values of themembership functions (the input values which result in an output valueof 0.5 in the membership functions) are weights Wc11 through Wc14 aswell as Wc21 through Wc24.

In other words, three kinds of membership functions are provided asdescribed earlier, and the center values of the respective membershipfunctions are coincident with the respective weights. For example, theweight of the center value of the membership function indicating “big”of the input value X1 is Wc11, the weights of the center values of themembership function indicating “middle” are Wc12 and Wc13, and theweight of the center value of the membership function indicating “small”is Wc14. Here, in the case of “middle”, two center values are obtainedbecause logical ANDs of the two membership functions are carried out.

Next, at the connecting sections between the former-half of themembership layer 22 and the latter-half of the membership layer 23,gradients of the membership functions are Wg11 through Wg14 and Wg21through Wg24. In this case, the gradients of the membership functionsare coincident with the respective weights. For example, the weight ofthe gradient of the membership function indicating “big” of the inputvalue X1 is Wg11, the weights of the gradients of the membershipfunction indicating “middle” are Wg12 and Wg13, and the weight of thegradient of the membership function indicating “small” is Wg14. Here, inthe case of “middle”, two gradients are obtained because logical ANDs ofthe two membership functions are carried out.

Finally, at the connecting sections between the rule layer 24 and theoutput layer 25, knowledge acquired from “expert” is represented byweights Wf1 through Wf9. In the knowledge acquired from “expert”, such arule weight of combination of input values that makes the output valuegreater is defined as a value closer to 1, and such a rule weight ofcombination of input values that makes the output value smaller isdefined as a value closer to 0. Other rule weights are initially definedas 0.5.

Moreover, the weights at the connecting sections other than theabove-mentioned connecting sections, such as the weights at theconnecting sections between the latter-half of the membership layer 23and the rule layer 24, are fixed to 1.

In the fuzzy neural network having the above-mentioned arrangement, amethod for finding the output values of the respective layers will bediscussed below. Here, since the output values of the input layer 21 arethe same as its input values, the explanation thereof is omitted.

As shown in the following equation (11), the membership layers add thecenter values Wc11 through Wc14 and Wc21 through Wc24 of the membershipfunction in its second layer.

Hi+j=Xi+Wcij  (11)

(Wcij<0; i=1, 2; j=1,2,3,4)

Here, X represents the output value of the input layer 21, Wc representsthe center value of the membership function, and H represents the outputof the second layer. Further, i represents the number of the respectiveinput items, and j is set to 1 in the case of “big”, set to 2 or 3 inthe case of “middle”, and set to 3 in the case of “small”.

The above-mentioned equation (11) indicates that the sigmoid functionshown in the following equation (12), which is to be substituted later,has the position of its origin set at the center value of the membershipfunction. $\begin{matrix}{{f(X)} = \frac{1}{1 + ^{- x}}} & (12)\end{matrix}$

Next, as shown in the following equation (13), in the third layer, bysubstituting multiplied gradients of the membership function into thesigmoid function, the output values of the membership function withrespect to the input values at the respective regions are obtained.Here, in the case of “middle”, instead of equation (13) the followingequation (14) is applied:

Mik=f(Hij·Wgij)  (13)

(k=1,2,3)

Mik=min{f(Hij·Wgij), f(Hi(j+!)·Wgi(j+1))}  (14)

Here, Wg is a value of the gradient of the membership function, f(X) isthe sigmoid function, M represents the output value of the membershipfunction, and min{f(X1), f(X2)} represents a logical AND of f(X1) andf(X2). Further, k represents a node number of the latter-half of themembership layer 23, and θ is set to 1 in the case of “big”, set to 2 inthe case of “middle”, and set to 3 in the case of “small”. Moreover, inthe above-mentioned equation (14), the smaller value is selected fromthe two functions inside the parenthesis of min by carrying out thelogical AND.

Next, in the rule layer 24, a calculation is carried out on the AND ruleby using the following equation (15): More specifically, with respect toeach of the two input items, one is selected from the three regions(big, middle and small), and a logical AND is carried out on the twomembership output values.

Rn=min{Mi1k1, Mi2k2}  (15)

(i<i; i=1, i=2; k, k=1, 2, 3)

Here, R is the output value of the AND rule, and k1 and k2 are nodenumbers of the latter-half of the membership layer 23. In this casealso, the smaller value is selected from the two functions inside theparenthesis of min by carrying out the logical AND.

Finally, in the output layer 25, the output value is calculated by usingthe following equation (16): In other words, calculations are carriedout as follows: the output values of the respective AND rules, obtainedby the preceding-item propositions of the fuzzy rule (for example, X1 isbig), are multiplied by the value of the weight Wf of the connectionresulted from the rule, and the resulting values are divided by thetotal value of all the outputs of the AND rules, and then the sum of theresulting values is found. $\begin{matrix}{y = \frac{\sum\limits_{n = 1}^{9}\quad \left( {{Rn} \cdot {Wfn}} \right)}{\sum\limits_{n = 1}^{9}\quad {Rn}}} & (16)\end{matrix}$

Here, n is a node number of the rule layer 24.

The above description shows calculating processes that are carried outfrom the time when input values were substituted in the fuzzy neuralnetwork to the time when the output values are obtained. Additionally,in the state where the fuzzy neural network having the above mentionedarrangement is first constructed, the values of the connections amongthe nodes of the respective layers are set to fixed values for therespective layers, and even if input values are substituted therein, theresulting output values are unreliable values, thereby failing toprovide a correct simulation of the input-output relationship of thetarget object. In order to obtain the correct simulation, it isnecessary to carry out adjustments on the weights of the connections.Therefore learning processes are required in the fuzzy neural network.

The following description will discuss the learning processes in theabove-mentioned fuzzy neural network.

Output values of sample data, which represent the input-outputrelationship of a target object, are used as teaching data T, and byusing the following equation (17), a squared error between the teachingdata T and each of the output values y that are obtained from the inputvalues (X1, X2, . . . Xn) of the sample data by using theabove-mentioned equations (11) through (16). $\begin{matrix}{E = {\frac{1}{2}\left( {T - y} \right)^{2}}} & (17)\end{matrix}$

Here, E represents the squared error between the teaching data T and theoutput value y. The correct simulation of the input-output relationshipof the target object is confirmed by making the error smaller.

One of the methods for decreasing the error is the back propagationmethod wherein a learning algorithm is used. The following descriptionwill discuss the learning algorithm:

When a partial differential is carried out on the above-mentionedequation (17) with respect to y, the following equation (18) isobtained; this equation indicates the effect on the output value y inrelation to the error. $\begin{matrix}{\frac{\partial E}{\partial y} = {- \left( {T - y} \right)}} & (18)\end{matrix}$

Next, when a partial differential is carried out on the above-mentionedequation (17) with respect to Wf, the following equation (19) isobtained. In this case, the aforementioned equation (16) is substitutedin y of the above-mentioned equation (17): $\begin{matrix}{\frac{\partial E}{\partial{Wfp}} = {{\frac{\partial E}{\partial y}\frac{\partial y}{\partial{Wfp}}} = {{- \left( {T - y} \right)}\frac{R}{\sum\limits_{n = 1}^{9}\quad {Rn}}}}} & (19)\end{matrix}$

Next, when partial differentials are carried out respectively withrespect to Wg and Wc on the above-mentioned equation (17), the followingequations (20) and (21) are obtained: In this case, the aforementionedequations (16), (15) and (14), or the aforementioned equation (13) and(11), are substituted in the above-mentioned equation (17).$\begin{matrix}{{\frac{\partial E}{\partial{Wgij}} = {{\frac{\partial E}{\partial y}\frac{\partial y}{\partial{Rp}}\frac{\partial{Rp}}{\partial{Mik}}\frac{\partial{Mik}}{\partial{Wgij}}} = {{- \left( {T - y} \right)} \cdot {\sum\limits_{r}{\left\lbrack \frac{{{Wfp}{\sum\limits_{n = 1}^{9}\quad {Rn}}} - {\sum\limits_{n = 1}^{9}\quad \left( {{Rn} \cdot {Wfn}} \right)}}{\left( {\sum\limits_{n = 1}^{9}\quad {Rn}} \right)^{2}} \right) \cdot {Mik} \cdot \left( {1 - {Mik}} \right) \cdot {Hij}}}}}}} & (20) \\{\frac{\partial E}{\partial{Wcij}} = {{\frac{\partial E}{\partial y}\frac{\partial y}{\partial{Rp}}\frac{\partial{Rp}}{\partial{Mik}}\frac{\partial{Mik}}{\partial{Hij}}\frac{\partial{Hij}}{\partial{Wcij}}} = {{- \left( {T - y} \right)} \cdot {\sum\limits_{r}{\left\lbrack \frac{{{Wfp}{\sum\limits_{n = 1}^{9}\quad {Rn}}} - {\sum\limits_{n = 1}^{9}\quad \left( {{Rn} \cdot {Wfm}} \right)}}{\left( {\sum\limits_{n = 1}^{9}\quad {Rn}} \right)^{2}} \right) \cdot {Mik} \cdot \left( {1 - {Mik}} \right) \cdot {Hgij}}}}}} & (21)\end{matrix}$

The above-mentioned equations (19) through (21) indicate the effect ofthe weight of each connection with respect to the error. Here, r in theabove-mentioned equations (20) and (21) represents the number of theoutputs of the AND rule selected from the weights that form membershipfunctions to be corrected. The sum of the errors from the nodes of therule layer 24 is found by using the number r.

The error as a whole can be reduced by correcting the weights in suchdirection as to decrease the effect. The amount of correction isrepresented by the following equations (22) through (24).$\begin{matrix}{{\Delta \quad {Wfp}} = {{- \alpha}\frac{\partial E}{\partial{Wfp}}}} & (22) \\{{\Delta \quad {Wgij}} = {{- \beta}\frac{\partial E}{\partial{Wgij}}}} & (23) \\{{\Delta \quad {Wcij}} = {{- \gamma}\frac{\partial E}{\partial{Wcij}}}} & (24)\end{matrix}$

Here, α, β and γ represent learning parameters, which are parameters fordetermining the amount of correction of the weight that reduces theeffect. Corrections as indicated by the following equations (25) through(27) are carried out using these parameters:

Wfp=Wfp+ΔWfp  (25)

Wgij=Wgij+ΔWgij  (26)

Wcij=Wcij+ΔWcij  (27)

The error is minimized by repeatedly carrying out learning processes inaccordance with the above-mentioned algorithms so as to correct theweights. Then, the learning processes are completed at the time when thevalue of the error has become not more than an acceptable value of theerror. The acceptable value of the error is predeterminately set, and inthe present embodiment, the learning processes are completed at the timewhen it becomes not more than 5%.

The resolution-converting process or the variable-magnification processis carried out on all partial images of an inputted image by using thefuzzy neural network that has been subjected to the learning processesas described above; thus, the inputted image is subjected to convertingprocesses, and a resulting image having the corresponding resolution orthe corresponding varied magnification is obtained in more appropriatemanner than the aforementioned neural network of the back-propagationtype.

In the image-processing method of the present embodiment, upon carryingout a resolution-converting process or a variable-magnification process,the hierarchical neural network is first subjected to a learning processby the existing-pixel-value learning circuit 6, using pixel values ofthe existing pixels inside matrixes that have been divided by theimage-data division circuit 4 as teaching data, as well as usingpositional information corresponding to the pixel values of the existingpixels as input data. After completion of the learning process, if theprocess in question is a resolution-converting process, interpolatedpixel values are obtained by inputting positional information of theinterpolated pixels, and if the process in question is avariable-magnification process, varied-magnification pixel values areobtained by inputting positional information of the pixels after thevaried magnification.

As described above, since the resolution-converting process and variablemagnification process are carried out on each of the divided matrixes ofthe input image by using a neural network of the back-propagation typeor a fuzzy neural network each of which is the hierarchical neuralnetwork and can execute a learning process for each input image, itbecomes possible to sharpen edge portions of the output image and alsoto make slanting lines thereof smooth without burs.

Moreover, since the hierarchical neural network can carry out a learningprocess on each input image, the input-output relationship can beoptimized whatever image may be inputted. Thus, since at least eitherthe resolution-converting process or the variable magnification processis carried out on each of the divided matrixes of the input image byusing the above-mentioned hierarchical neural network which can executea learning process on a real-time basis, the weights of the network isadjusted for each input image; thus, it becomes possible to always carryout an optimal converting process.

Therefore, irrespective of images to be inputted, it is possible toeliminate the blurred state of the edge portions and burs in theslanting lines which are problems caused by the converting processes,and consequently to make the image after the conversion smooth, therebyimproving the quality of the converted image.

Moreover, the hierarchical neural network in the above-mentionedimage-processing method is designed so that positional information ofthe respective pixels of an input image is used as inputs and pixelvalues corresponding to the inputted positional information of thepixels are used as outputs.

As described above, upon carrying out the resolution-converting processand the variable-magnification process with respect to an input image,the hierarchical neural network only requires positional information ofthe respective pixels of the input image as its inputs, and theresulting outputs represent pixel values at respective positions insidethe matrixes of the input image; therefore, it is not necessary toprovide a complicate input-output relationship in the neural network,and therefore, it is possible to apply a neural network of small scalewith a simple construction.

Moreover, in the present image-processing method, upon carrying out theresolution-converting process and the variable-magnification process,the learning process of the hierarchical neural network is executedbased upon the positional information of the existing pixels and theexisting pixel values; therefore, only a small amount of calculations isrequired for the learning process, and a high-speed learning process isobtained. Thus, it becomes possible to carry out theresolution-converting process and the variable-magnification process ofan image at high speeds.

Furthermore, since a neural network of the back-propagation type is usedas the hierarchical neural network in the image-processing method, theconverting process of an input image can be carried out by using asimple hardware construction, upon carrying out theresolution-converting process and the variable-magnification processwith respect to the input image.

In the case when a fuzzy neural network with two inputs and one outputis used as the hierarchical neural network, upon carrying out theresolution-converting process and the variable-magnification processwith respect to an input image, it becomes possible to provide a moredetailed gradation curved surface representing the partial images,although a more complex hardware construction is required compared withthe above-mentioned neural network of the back-propagation type, andconsequently to provide more optimal interpolated pixel values orvaried-magnification pixel values than those of the neural network ofthe back-propagation type.

Additionally, in the present embodiment, the neural network of theback-propagation type shown in FIG. 3 and the fuzzy neural network shownin FIG. 5 are used as the hierarchical neural network; however, thepresent invention is not intended to be limited by these networks, andanother hierarchical neural network having only connections that aredirected from the input layer to the output layer may be applied toobtain the same effect.

The invention being thus described, it will be obvious that the same maybe varied in many ways. Such variations are not to be regarded as adeparture from the spirit and scope of the invention, and all suchmodifications as would be obvious to one skilled in the art are intendedto be included within the scope of the following claims.

What is claimed is:
 1. An image-processing method comprising: a firststep of dividing an input image having n(n>1) gray scales into aplurality of matrixes; a second step of carrying out at least either aresolution-converting process or a variable magnification process foreach of the divided matrixes, by using a hierarchical neural networkthat can execute a learning process for each input image at real time;wherein said second step includes the sub-steps of: subjecting thehierarchical neural network to a learning process, using pixel values asthe existing pixels inside matrices as teaching data, as well as usingpositional information corresponding to the pixel values of the existingpixels as input data, and after completion of the learning process, ifthe process in question is a resolution-converting process, obtaininginterpolated pixel values by inputting positional information of theinterpolated pixels, and if the process in question is avaried-magnification process, obtaining varied-magnification pixelvalues by inputting positional information of the pixels after thevaried magnification; and a third step of outputting the image processedin the second step as an output image having n gray scales.
 2. Theimage-processing method as defined in claim 1, wherein the hierarchicalneural network is designed so that positional information of therespective pixels of an input image is used as inputs thereof and pixelvalues corresponding to the inputted positional information of thepixels are used as outputs thereof.
 3. The image-processing method asdefined in claim 2, wherein the hierarchical neural network is a neuralnetwork of a back-propagation type including an input layer constitutedby two nodes, an intermediate layer constituted by more than one nodeand an output layer constituted by one node.
 4. The image-processingmethod as defined in claim 2, wherein the hierarchical neural network isa fuzzy neural network including an input layer constituted by twonodes, membership layers that are two layers forming membershipfunctions respectively representing “big”, “middle” and “small”, a rulelayer that makes combinations of all membership values with respect totwo inputs so as to obtain fuzzy logical ANDs, and an output layerconstituted by one node.
 5. An image processing apparatus comprising: animage-data input device that reads an original image as image datahaving n(n>1) gray scales; an image-data division circuit that dividesthe image data that has been read by the image-data input means intomatrixes in accordance with positional information of respective pixels;an image-data conversion device that carries out at least either aresolution-converting process or a variable magnification process foreach of the matrixes divided by the image-data division means, by usinga hierarchical neural network that can execute a learning process foreach of the image data at real time; wherein the image-data conversiondevice includes: an existing pixel-position-information/pixel-valueextracting circuit that extracts positional information of each ofexisting pixels contained in the image data that has been divided intothe matrix and a pixel value at each position, as input data andteaching data for a hierarchical neural network, an existing pixel valuelearning circuit that gives the input data and teaching data that havebeen extracted by the existing pixel-position-information/pixel-valueextracting circuit to the hierarchical neural network, thereby allowingthe neural network to learn, a converted-pixel-position input circuitthat inputs positional information of the respective pixels after theconverting process to the hierarchical neural network that has beensubjected to the learning process in the existing pixel value learningcircuit, a converted-pixel-arithmetic circuit that calculates andobtains pixel values at the respective positions by using thehierarchical neural network to which the positional information of therespective pixels has been inputted by the converted-pixel-positioninput circuit, and a converted-pixel-value output circuit that outputsthe converted pixel values that have been found by theconverted-pixel-value arithmetic circuit to an image-data output deviceas image data after the converting process, and wherein the image-dataoutputting device outputs the image processed by the image-dataconversion device as an output image having n gray scales.
 6. Theimage-processing apparatus as defined in claim 5, wherein the image-datadivision means divides the inputted image data into matrixes, thelongitudinal and lateral sizes of each matrix being set to divisors ofthe longitudinal and lateral pixel numbers of the input image data. 7.The image-processing apparatus as defined in claim 5, wherein thelearning means allows the hierarchical neural network to finish thelearning processes at the time when a predetermined number of learningprocesses has been carried out.
 8. The image-processing apparatus asdefined in claim 5, wherein the learning means allows the hierarchicalneural network to finish the learning processes at the time when thelearning error has reached a predetermined value of error.
 9. Animage-processing method comprising: a first step of dividing an inputimage having n(n>1) gray scales into a plurality of matrixes inaccordance with positional data of respective pixels; a second step ofobtaining pixel values of the respective pixels by using a hierarchicalneural network that can execute a learning process for each input imageat real time; a third step of using the pixel values obtained in thesecond step as teaching data for the hierarchical neural network, saidthird step including subjecting the hierarchical neural network to alearning process, using pixel values as the existing pixels insidematrices as teaching data, as well as using positional informationcorresponding to the pixel values of the existing pixels as input data;a fourth step of carrying out at least either a resolution-convertingprocess or a variable magnification process for each of the dividedmatrixes, by using the hierarchical neural network, wherein, aftercompletion of the learning process, if the process in question is aresolution-converting process, obtaining interpolated pixel values byinputting positional information of the interpolated pixels, and if theprocess in question is a varied-magnification process, obtainingvaried-magnification pixel values by inputting positional information ofthe pixels after the varied magnification; and a fifth step ofoutputting the image processed in the fourth step as an output imagehaving n gray scales.
 10. The image-processing method as defined inclaim 2, wherein the third step includes a first substep of using theteaching data to obtain output values, by using the hierarchical neuralnetwork, a second substep of comparing the output values obtained in thefirst substep with the pixel values obtained in the second step toobtain an error value, a third substep of varying weights at connectionsin the hierarchical neural network for reducing the error value obtainedin the second substep, and repeating the first, second, and thirdsubsteps for minimizing the error value obtained in the second substep,the first, second and third substeps being repeated until the errorvalue obtained in the second substep is less than or equal to apredetermined value.